Developing a Usability Evaluation Method for E-learning Applications: From Functional Usability to Motivation to Learn

In this paper the development of a questionnairebased usability evaluation method for e-learning applications is described. The method extends the current practice by focusing not only on cognitive but also affective considerations that may influence e-learning usability. The method was developed according to an established methodology in HCI research and relied upon a conceptual framework that combines web and instructional design parameters and associates them with the most prominent affective learning dimension, which is intrinsic motivation to learn. The latter is proposed as a new usability measure that is considered more appropriate to evaluate e-learning designs. Two large empirical studies were conducted in order to evaluate usability of e-learning courses offered in corporate environments. Results provide significant evidence for reliability and validity of the method; thus usability practitioners can use it with confidence when evaluating the design of elearning applications. 1. The need to develop a usability evaluation method for e-learning applications Organizations and educational institutions have been investing in information technologies to improve education and training at an increasing rate during the last two decades. Especially in corporate settings continuous education and training for the human resource is critical to an organization’s success. Electronic learning (e-learning) has been identified as the enabler for people and organizations to keep up with changes in the global economy that now occur in Internet time. Within the context of corporate training, e-learning refers to training delivered on a computer that is designed to support individual learning or organizational performance goals (Clark and Mayer, 2003). Although e-learning is emerging as one of the fastest organizational uses of the Internet (Harun, 2002), most e-learning programs exhibit higher dropout rates when compared with traditional instructor-led courses. There are many reasons that can explain the high dropout rates such as relevancy of content, comfort level with technology, availability of technical support etc. but one major contributor is the poor usability of e-learning applications. The latter is the focal point of this study. Evaluating the usability of e-learning applications is not a trivial task. Increase in the diversity of learners, technological advancements and radical changes in learning tasks (learner interaction with a learning/training environment is often an one-time event) present significant challenges and render the possibility of defining the context of use of elearning applications. Identifying who are the users and what are the tasks in e-learning context impose extra difficulties. In the case of e-learning design the main task for the user is to learn, which is rather tacit and abstract in nature (Zaharias and Poulymenakou, 2006). As Notess (2001) argues “evaluating e-learning may move usability practitioners outside their comfort zone”. Squires (1999) highlights the need for integration of usability and learning and points out the noncollaboration of workers in HCI and educational computing areas. In fact usability of e-learning designs is directly related to their pedagogical value. An e-learning application may be usable but not in the pedagogical sense and vice-versa (Quinn, 1996, Albion, 1999, Squires and Preece, 1999). Accordingly usability practitioners need to familiarize themselves with the educational testing research, learning styles and the rudiments of learning theory. Nevertheless very little has been done to critically examine the usability of elearning applications; there is an ellipsis of research validated usability evaluation methods that address the user as a learner and consider cognitive and affective learning factors that support learners to achieve learning goals and objectives. 1.1 Beyond functional usability: the emergence of affective dimension A major challenge of contemporary HCI research is to address user affect. It is critical that systems designers assess the range of possible affective states that users may experience while interacting with the system (Hudlicka, 2003). To this end new usability techniques and measures need to be established (Hornbaek, 2005). Traditional usability measures of effectiveness, efficiency and satisfaction are not adequate for new contexts of